Method and apparatus for creating a virtual microscope slide

ABSTRACT

A method and apparatus are disclosed for making and using a virtual microscope slide data structure having data for an overall image of the specimen or a substantial portion thereof along with data for higher magnification images of selected areas within the overall image and along with data useable for correlating, linking and/or coherently assembling low magnification, overall image segments with the higher image segments. This latter data may comprise a control program for manipulating images to allow zooming different, higher magnification images into view, scrolling adjacent high magnification images into view or marking the location of a displayed higher magnification view on the overall image to assist in navigation by a viewer to locate and quickly display suspicious areas within the specimen. The data structure may be sent over the Internet.

FIELD OF THE INVENTION

This patent application is a continuation patent application of patentapplication Ser. No. 10/365,548, filed on Feb. 12, 2003, now U.S. Pat.No. 6,775,402 which is a continuation patent application of patentapplication Ser. No. 09/574,423, filed on May 19, 2000, now U.S. Pat.No. 6,522,774, which is a continuation of patent application Ser. No.09/032,514, filed on Feb. 27, 1998, now U.S. Pat. No. 6,272,235, whichis a continuation-in-part of patent application Ser. No. 08/805,856,filed on Mar. 3, 1997, now U.S. Pat. No. 6,101,265.

BACKGROUND OF THE INVENTION

The invention described in the aforesaid application answers a need, arequirement to image and digitally record an object in a relatively flatplane at high resolution/magnification. Today, it is impractical toconstruct an optical image sensor large enough to cover the entire imagearea e.g., of a specimen on a microscope slide, at the requiredresolution. This is because lens size and resolution/magnificationissues limit the size of the field of view of magnified objects andtheir resulting images. Viewing through a microscope is akin to viewingthrough a periscope in that one sees a very small field of view even atlow magnifications, such as 1.25×. A pathologist using a microscopeoften scans a slide to obtain in his mind an overall view or sense ofwhat constitutes the specimen and he remembers the general locations ofthe diagnostically significant, small pieces of the specimen. Usually,these are the diseased areas, such as malignant or potentially malignantportions of the specimen. To obtain higher resolution and magnificationof these suspicious portions, the pathologist switches to a highermagnification objective lens but then the field of view becomes muchsmaller again. Often, the pathologist switches back and forth betweenthe lower magnification, larger field of view objective lens to orienthimself relative to the specimen and the high magnification, smallerfield of view to obtain the detailed, high resolution view of thesuspicious area on the specimen. Thus, the user never receives amagnified, condensed overall view of the specimen or a portion of thespecimen but must remember the series of views taken at lowmagnification. Likewise, at high resolution, high magnification, theuser never receives or views a collection of adjacent images but mustinterrelate these successive images in the user's mind.

A similar problem exists on the Internet or intranet where a pathologistmay receive a single field of view magnified image taken from a specimenover the Internet or the intranet on his browser. The pathologist mustbe provided with explanations to coordinate the high resolution viewwith the lower resolution view. The number of views available to thepathologist is very limited, and the pathologist is unable to selectother views or to scroll to neighboring views at the areas that are mostinteresting to the pathologist.

In the aforesaid prior application, there is disclosed a method andapparatus whereby a person may construct a low magnification, digitizedoverall, image view of the entire specimen on a slide or a selectedportion of the specimen on a slide, such as the basal layer of a tissuesection. The overall, low magnification digitized image allows the userto understand where the user is presently located in his viewing andwhere the user may want to make the next observation. That is, the lowmagnification overall view is generally in color and provides to theexperienced user a visual overall or thumbnail view of the slide andshows the possible areas of interest for malignancy or other diseaseswhich manifest themselves at certain locations on the specimen imagebeing viewed. This low magnification overall view enables the user toselect thereon the points of interest that the user wants to view at ahigher magnification.

The overall view was constructed by taking by a large number of lowmagnification images of the specimen through a microscopic scanningsystem and then coherently assembling and coordinating these respectivesmaller views or images (hereinafter “image tiles”) into one coherent,low magnification, macro image from the specimen. Often the digitizedmacro image is reduced in size by a software system to even a smallersize to be displayed on a local screen or to be transferred over a lowbandwidth or a high bandwidth channel to a remote viewing screen.

The prior application teaches how to assemble a large number of imagetiles, for example, 35 image tiles for the macro image, and then to takea series of other tiles of a higher magnification or magnificationswhich will also be viewed by the user. To this end, the user is providedwith a marker, such as a cursor or the like, to select the defined areaof interest, and by a simple command, to cause the selected, highermagnification digitized images to appear on the screen for viewing bythe user. The higher magnification images may be one of severalmagnifications or resolutions such as 10×, 20× and 40×.

As disclosed in the aforesaid application, it is preferred to allow theuser, such as a pathologist, to quickly flip back and forth between thehigh resolution micro image and the low macro resolution image or toprovide separate split screens whereby the pathologist is shown anoverall macro view and a marker showing where the current highermagnification view is located. Because of the multiple magnifications,the user may change to an intermediate magnification such as would beaccomplished by switching between intermediate objective lenses. Thisprovides the pathologist with views which correspond to changing backand forth between objective lenses in a microscope, a procedure withwhich most pathologists are familiar and have been trained.

Additionally, the aforesaid application provides the user with ascrolling feature that allows the user to shift into the viewing screenadjacent, magnified images on the screen so that the pathologist is notlimited to only seeing just a full tile view but may see adjacent imagematerial from adjacent, neighboring tile images.

In the aforesaid patent application, there is a disclosure oftransmitting the low magnification image over a local area network orover the Internet through various servers and computers. The tiledimages that were being transmitted were achieved by use of a fullycomputer controlled microscope which allowed the user to navigate alonga specimen area of interest, such as along a basal area or to othersuspicious points spread throughout the specimen to acquire tiled imagesof selected areas so that the entire specimen would not have to bedigitized and stored. As disclosed in the preferred embodiment in theaforesaid application, an Internet browser remotely-controlled,automated microscope could be used by a pathologist from a remotelocation to view the reconstructed macro image tiles; and, with hismanipulation of the microscope, using an intranet or Internet browser,could acquire single images at higher magnifications if desired. Whileseveral people could see the particular digitized images beingtransmitted out over the Internet as they were acquired by a particularpathologist and several people could view the stored images, there wasstill a problem of control at operation of the microscope by each personviewing the digitized images, and a problem with acquiring andtransmitting large areas of higher magnification images using the tilingmethod.

As stated above in greater detail, the current state of archiving thedigital images achieved through a microscope is often by havingphotographs or by video tapes. The photographs are difficult to use asis a video tape particularly when the user wants to move rapidly backand forth between various images and to scroll through various adjacentparts of the specimen image. Further, current archival methods lack anoverall macro image of the specimen, which allows the user to knowexactly where the particular high resolution view is from when it ismaking an analysis of the high resolution image.

While digitized images can be stored magnetically or otherwise digitizedand recorded on various recording mediums, no current archival systemallows the user to toggle back and forth between high magnificationimages and low magnification images or between various images atdifferent magnifications such as that achieved by a pathologistswitching microscope objective lenses in real time to get the macro andmicro images from the same location on the specimen. Heretofore, thepractice of pathology has been relatively limited to the use ofmicroscopes and to the pathologist having to use the microscope toreview the particular specimen.

There is a need for a dynamic system whereby one or more or severalpathologists, including a consulting pathologist, may view the same areasimultaneously and interact with one another either in diagnosis or inanalysis. Also, it would be best if the images from the specimen couldbe stored so that a pathologist could easily examine the images at hisleisure using an intranet or Internet browser at a later date merely byaccessing the particular web site where the images are located.

It will be appreciated that a host of problems need to be solved toallow Internet or intranet users to view on their respective monitorsuseful, low resolution, macro images and high resolution, micro imagesof several adjacent, original microscope images. One of the firstproblems is how to seam together neighboring tile images to form aseamless overall view of these tiles. Heretofore, attempts to seam thetiles used software to combine the pixels at the tile boundaries andhave been generally unsuccessful. Another problem is that of mapping ofcoordinates beginning with the coordinates, usually X and Y coordinates,from and at the microscope stage carrying the slide and then the mappingof coordinates on the scanning screen not only for one magnification butalso to coordinate the mapping for the respective multiple resolutionimages taken typically at 1.25×, 10× and 40× or more. These coordinatesmust be maintained for a large number of tiled images, e.g., 40 tiledimages for one macro image. In order for the remote user to view thesetile images and to flip back and forth between different resolution,tiled images, the user's computer and monitor not only must receive theaddresses and stored parameters for each pixel but must also run them ona generic viewing program.

Another problem with acquiring image tiles and sending them over a lowbandwidth Internet channel is that both the storage requirements on theserver and the amount of data acquired per slide become high, such asfor example, 120 megabytes to one gigabyte. The 120 megabytes is onlyachieved by not taking image tiles of the entire specimen but only imagetiles from the areas selected by the pathologist when tracing at highresolution along basal layers or only at the dispersed, suspiciouscancer appearing area in a breast cancer. Even with this selectiveinteraction by a pathologist in constructing the macro and microdigitized images with a vastly reduced amount of image tiles relative tothat which would be acquired if the entire specimen where imaged at eachof the multiple magnifications, the acquired amount of data is amonstrous problem of transmitting in a reasonable amount of time over anarrow bandwidth channel to an ordinary web browser having limitedstorage capacity. While rough compression techniques could be used, theycannot be used at the expense of providing the high resolution imagethat the pathologist must have for diagnosis of the specimen.

SUMMARY OF THE INVENTION

In accordance with the present invention, there is provided a new andimproved method and apparatus for constructing digitally scanned imagesfrom a microscope specimen, for storing the digitally scanned images ina tiled format convenient for viewing without a microscope, and fortransferring the tiled images at multiple magnifications for viewing byanother at a remote location. This is achieved by assembling togetherseveral adjacent, original microscope views at a first magnification toobtain an overall macro view of the specimen and assembling togetherseveral adjacent original microscope views at a higher magnification tocreate a combined data structure. The data structure may then betransferred to the remote viewer to provide this viewer multipleresolution macro and micro images of areas on the slide specimen. Thedata structure is constructed by digitally scanning and storing the lowmagnification images with their mapping coordinates and likewise,digitally scanning and storing higher magnification images with theirmapping coordinates. Further, a pathologist may interactively selectonly those diagnostically significant areas of the specimen for digitalscanning and storing to reduce significantly the number of image pixelsstored at high resolution.

The data structure can be transmitted over the Internet or intranet toallow multiple users to consult on a particular microscope each usinghis own virtual images of the specimen. These users each may flip backand forth between different resolution images in a manner similar tothat achieved when shifting among objective lens for differentresolution views. However, the preferred embodiment of this inventionprovides a marker on the overall macro view showing the remote userwhere the higher resolution image is located on the specimen so that theuser does not have to remember the location of the higher resolutionimage. Unlike the single, small optical field of view currentlyavailable, the remote user is provided with a series of abutted, tiledimages each being substantially equal to one small optical field ofview. Thus, the remote user is provided with better and larger macro andmicro tiled images than the single, small optical fields of view takenat the same magnifications of a single tiled image.

The preferred data structure is also provided with a generic viewingprogram that allows the remote user to manipulate and interpret thetiled images on the. user's browser. This generic viewing program isself-contained with its own display and the interpretative program isusable with a variety of computers, browsers and monitors. The datastructure uses selectively compressed data to reduce the huge amount ofacquired data, e.g. 120 megabytes, into a small amount of data, e.g. 1.4megabytes. Such smaller, more manageable amounts of data can betransmitted over a low bandwidth channel such as the Internet withoutthe loss of resolution that would interfere with the remotepathologist's analysis. Further, the interactive program allows thepathologist to scroll and to view neighboring image areas of neighboringimage tiles which were currently unavailable to the pathologist untilthe inventions set forth in the aforesaid application and in thisapplication.

Turning now in greater detail to aspects of this invention, problemswith achieving tileable (i.e. contiguous images which can be seamlesslyabutted next to each other to recreate the original image, but atdifferent magnifications) multiple images of a specimen on a microscopeslide are overcome by the system of the invention. The system includes amicroscope and stage in which digital locations on the stage have beenpredetermined in accordance with an electromechanical addressablecoordinate system (X-Y for convenience). Each point on the stage isassigned an “X” and a “Y” coordinate which uniquely defines itslocation. The increments in each of the X and Y directions areestablished at a predefined amount for example in 0.1 micrometerincrements. A key factor in achieving superior resolution of thespecimen images at higher magnifications is to establish many morephysical increments on the stage for each pixel of the image sensor andof the intended display. For example, at 1.25× magnification, 64 pointson the stage correspond to one pixel on a CCD optical sensor, whichcorresponds to one pixel on a 640 by 480 monitor (for a VGA display),using the bitmap addressing and scrollable image method describedherein.

Once the coordinate system is defined for the microscope stage, when aspecimen on a microscope slide is placed on it, each feature of intereston the slide can be uniquely located with reference to the stage. Thenthe microscope system is used to digitally scan the image. The firstscan is done at a relatively small magnification since this image willbe used to provide a “macro” image of the entire specimen. In thepreferred embodiment, 1.25× magnification is used. The microscope systemthen scans the slide using the 1.25× objective. Since the image isdetected by rectangular optical sensors, such as the optical sensors ina CCD grid, the stage must be moved in relatively larger increments toplace the next adjacent physical part of the slide exactly in the regionwhere that rectangular area will be precisely imaged on the CCD sensor.

Although the area traveled is relatively large, the precision must behigh to enable alignment of the image parts within the pixel resolutionof the CCD sensor. For example, at the 1.25× magnification, 48,143 Xsteps and 35,800 Y steps are necessary to move the specimen object onthe stage to a new, contiguous region for optical imaging on the CCDsensor. The signal produced by the optical sensors in the CCD grid arethen transmitted to a computer which stores the image signals in aseries of tiled images. Since each image frame is defined bypredetermined X-Y coordinates, these can be easily converted into aseries of contiguous tiled images.

To view the scanned digital image on a monitor, the computer uses amethod of reserving an image bitmap corresponding to the entire size ofthe tiled image, e.g., in this instance, 10×8, 1.25× magnified tiledimages are acquired. This requires an image bitmap of 7,520×3,840 insize, using a 752×480 pixel CCD sensor. Since the X-Y coordinates areknown for each image tile, and thus for each pixel in each tile, thebitmap can be used to coordinate and display the stored image tiles topresent a fused macro view of the image with one-to-one pixelcorrespondence of the screen pixels with the image pixels. Typically,the screen pixels are fewer in X-Y size than the macro tiled image,(that is, the entire image cannot be viewed on the monitor without somesort of image compression); and in this case, the macro tiled image isscrolled on the viewable window segment of the screen to maintain theone-to-one correspondence. An advantage of the one-to-one correspondenceis that significant image detail is available to the user. Further,since the physical X,Y position on the specimen is known through thestage coordinate relationship to the image pixels, the tiled macroimages can be used to locate regions, and move the stage to that regionfrom collection of higher magnification tiled images.

Since the nature of optics, i.e. lenses, is that they provide agenerally circular image with a sharp central region and with fuzzinessaround the periphery of the image, the microscope system is designed tostep through the various locations on the slide in such a manner to scanonly the high resolution image portion in the center of the opticalimage. The fuzzy outer regions are discarded. This also has the benefitof ensuring a high resolution image once the tiled images arereconstructed for viewing by a user on a monitor.

After the macro image is completed, a trained professional, such as anexamining pathologist, views the image of the specimen by viewing themacro image and looking for areas of interest. In general, most specimenslides contain only a few small areas of diagnostic significance. Thebalance of the slide is generally empty or not significant. When theexamining pathologist views the slide, some areas may have beenpreviously marked in the regions of interest for viewing and analysis athigher magnifications. Once these regions are marked, the microscope isset to the desired higher magnification and then only the marked regionsare scanned and stored. Alternatively, he may define new areas directlyon the macro image. In either case, the regions are outlined using apointing device, such as a mouse, directly on the viewing windowdisplaying the macro image. As described above with respect to the 1.25×images, since the stage has a predefined coordinate system, the scannedhigher magnification image portions can be easily located with respectto the macro image, creating a series of micro images.

The fact that a typical microscope specimen slide contains only limitedinformation of interest and the ability of the system embodying theinvention to accurately locate such regions enables the system to createa virtual microscope slide, i.e. a data structure which can be used inplace of the actual specimen slides. This advantageously enablesmultiple users to consult on a particular specimen. Additionally,because of the reduced size of the data structures, they can be viewedlocally on a personal computer, transmitted over an intranet or via theInternet globally. The created data structures can be stored on avariety of storage or recording media: for example, on a server's harddisk, a Jazz drive, a CD-Rom or the like. Storing the data structure ona portable storage media further enables the transfer and archiving ofthe microscopic slide data structures by multiple users.

Another feature of the invention is a self-executing data structure.This is achieved by packaging the tiled images with an active, dynamiccontrol program. When an active dynamic control program is used by aviewing program such as a common web browser, the browser can interpretthe dynamic control program. This allows the user to interact andcontrol the viewed images seen on the viewer's screen from the recordingmedium. More specifically, in the preferred embodiment of the invention,a large number of low magnification, digitized tiled images are formedand embedded in a data structure with linking information allowing themto be coherently tiled to each other during viewing to form a macroimage, and a series of higher magnification tiled images also similarlyconstructed into a micro image, and a control program such as a JAVAapplet, is provided and transferred with the macro and micro tiledimages for use by a remote user. Thus, for example, the macro and microtiled images with their active control program may be transmitted overan Internet or an intranet to a browser, or other application programfor viewing the images, or other application program for viewing theimages, where the user may then access the browser to analyze the imagesat multiple resolutions and with a macro field of view before the user.This enables the viewing of the images in a manner similar to the use ofan optical microscope, but in this case visually the view is of avirtual microscope slide at multiple resolutions.

Also, in accordance with the invention, the constructed, tiled macro andtiled micro images along with the control program can be placed on a webserver and can be accessed locally and over a wide area, even globally,by multiple users at various times. For instance, a large number ofpreviously scanned and recorded specimen slides, such as 300 specimenslides, may have their respective micro and macro tiled images put on aserver. Medical students or pathology students then may each access theslide or all of the 300 slides and review them on their respective webbrowsers at their leisure. Likewise, a pathologist may dial up orotherwise connect through an internet service provider to the Internetor other long-level network and access a web server and obtain aparticular patient's specimen results. Those results would have beenstored as a data structure (including macro and micro tiled images alongwith the control and interpretative program). The pathologist then mayand perform an analysis at his home or in his office without needing tohave or to control a microscope or the particular slide. The pathologistmay toggle back and forth between the micro and macro images, and thendictate or otherwise prepare his analysis, findings or diagnosis fromthese stored images. This advantageously enables the pathologist toperform part of his job in the convenience of his home or office andalso enables a laboratory to maintain actual specimen slides in a safeand secure location, away from the potential of damage and without thenecessity of shipping the slides for microscopic examination at a remotelocation.

The control program, which in the preferred embodiment of the inventionis a dynamic self-executing program such as a JAVA applet, allows theuser to manipulate and interpret the images while on a browser. Thedynamic, self-executing program is completely self-contained with itsown display and interpretative program for operation by the user of thebrowser.

The present invention is not limited to use on a browser since thetiled, digitized images and the active, control program may be stored ona CD-ROM or other portable storage medium and sent through the mail, orotherwise transferred to the user for review at the user's conveniencewith dedicated viewers.

Thus, from the foregoing, it will be seen that there is provided a newand improved method of and apparatus for archiving of microscopic slideinformation on a storage medium with an active control program, whichallows the display and interpretation of various micro and macro images.

In accordance with another important aspect of the invention, there isprovided with the self-executing data structure (the stored macroimages, micro images and dynamic, self-executing program for viewing,reconstructing and manipulating the stored images) the ability to scrollthrough the displayed images. This allows the user not only to see oneimage tile at a particular magnification, but also to use a pointer orto otherwise move a point to cause displayed images from adjacentneighboring image tiles which were not previously viewable to beincluded in the field being viewed by the user. That is, the user mayshift the viewing location across tile boundaries from one tile toanother, and up or down, or right or left or to other points of interestin a normal two-dimensional scrolling manner. Thus, the user is providedwith an archived stored slide at multiple magnifications which can bereadily scrolled through in any arbitrarily chosen direction ordirections. As in the aforesaid application, the user interactively willgo to various areas of selected interest and operate a pointer or amarker to select for high magnification viewing the particular area ofinterest and also do a scrolling of neighboring areas of interest.

In addition to the Internet browser, the data images can be viewed,reconstructed and manipulated using a dynamic, self-executing programsuch as, for example, a JAVA applet or an ACTIVE-X applet. An advantageof using a dynamic, self-executing program which is linked with the dataimages on a data structure is that the data images can be viewed,reconstructed and manipulated independent of the operating system of theusers computer. Additionally, the user does not have to acquire thelatest version of the dynamic, self-executing program since it isalready linked with and provided with the data images on the datastructure or on the storage medium. Thus, the user can always view thedata images, regardless of different program versions.

The dynamic, self-executing program permits interchanging the image inits entirety simulating the visual effect of changing objectives in aregular, mechanical optical microscope view. Thus, the user can easilyswitch from one magnification to another and scroll through portions ofthe image, simulating tracking the image by moving the slide under themicroscope lens.

The dynamic, self-executing program permits scrolling the image in awindow to enable viewing of the reconstructed large field of viewimages. The user can use a mouse, or other pointing device, to select aportion of the image on the large field of view image and the programwill display that selected portion in another window at the desiredmagnification.

A method of constructing a record of the digital image of a specimen ona microscope slide using image tiles includes scanning the image at afirst low magnification so that substantially all of the specimen isobtained. Then the specimen is scanned at a second higher magnificationso that images of selected (or all) sub-portions of the specimen areobtained. The spatial relationships of the first lower magnificationimage to the second higher magnification images is used to reconstructthe image during viewing. The individual, sub-portions or tiles of thescanned image are seamed together by the dynamic, self-executing programto create a digital image of substantially larger areas thanindividually acquired image fields of view without tiling.

A data structure according to the invention is created by firstdigitally scanning the desired specimen at a plurality of imagemagnifications. The scanned images are then stored in a series ofcontiguous image tiles. Then the stored images are linked with adynamic, self-executing program. The data structure can be created usinga software program. Images are preferably first stored as bitmap files(.bmp). (Note that storing the resulting image files in the bitmapformat is different from the bit mapping method of creating the imagefiles described herein.) An image compression program is used to convertthe bitmap files to a JPEG (.jpg) format, which requires less storagespace and consequently less time to display on a computer. The personcreating the data structure can select how much detail to include in theconversion. JPEG images can be created for example, using 20 to 80%compression ratios of the original image. An advantage of the JPEGformat is that essentially empty tiles (tiles with mostly white or blackspace) compress down to very small files. Detailed files, however, donot compress as much. Additionally, the dynamic, self-executing programmay include compression algorithms for displaying the entire image orportions thereof in the viewing window.

After downloading or installation of a data structure on a storagemedium, when the user desires to view the data images, he uses a mouseand “clicks” on the icon for the self-executing data structure. Thedynamic, self-executing program displays the image in a window.Typically, the program will display a macro or thumbnail view of theentire specimen image at a lower magnification and a smaller windowcontaining a particular image tile or groups of tiles at a highermagnification. The program enables the user to use the mouse or otherpointing device to select a point or outline a region on the thumbnailview. The selected view will then be displayed in the smaller window atthe second magnification. The user can move the mouse or pointing deviceand the image in the smaller window will scroll with the selection onthe thumbnail view. In this way, the program simulates movement of amicroscope slide under the field of view of the mechanical microscope.However, it should be noted that because of the one-to-onecorrespondence between the CCD pixels and the screen pixels, not allmacro images may be able to be displayed on the monitor. The user mayscroll through the macro image or select a compression feature todisplay the entire macro image in the window.

Another feature of the self-executing data structure is that when theimage is displayed on the viewing screen, the user can select an imagetile or sub-portion of the image and alternately view that portion ofthe image at each scanned magnification. For example, if the data wasscanned at magnifications of 1.25×, 20× and 40×, the user can “click”and see the same tile at each of those magnifications alternately.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system according to the invention forcreating and transmitting locally, over an intranet or via the Internetdata structures of an image of specimen on a microscope slide;

FIG. 1A is representation of a microscope slide which has beenarbitrarily assigned to be scanned into eighty tiled images;

FIG. 1B is a representation of the detected signals of the individualpixel sensors in a CCD optical array after detecting a selected imagearea to tile and the referenced data files containing the informationdescribing the detected signals;

FIG. 2 is a screen view of a system embodying the present inventionshowing a low magnification image of a specimen on a microscope slide inone window, a high magnification image of a portion of the lowmagnification image selected by a region marker and a control window;

FIG. 3 is a view of a display screen of the apparatus embodying thepresent invention showing the control window a low magnification windowhaving a plurality of high magnification micro image regions delineatedtherein and a high magnification window including one or more of themicro image regions;

FIG. 4 is a view of a macro image of an actual breast cancer specimendisplayed at 1.25× as seen on a computer monitor;

FIG. 5 is a view of the grid portion of FIG. 4 outlining a region ofinterest selected by a pathologist displayed at 40× magnification;

FIG. 6 is a block diagram of the steps in the mapping of the scannedimage from the optical sensor array to computer bit map in memory to thedisplay on a user's monitor;

FIG. 7A is a file listing such as would be seen under Windows 95 filemanager showing the data files included in a data structure for a breastcancer specimen;

FIG. 7B is a file listing of a Java applet for controlling a datastructure;

FIG. 8 is file listing such as would be seen under Windows 95 filemanager showing the data files included in an alternate data structurefor a breast cancer specimen;

FIGS. 9A and 9B are a block diagram of the apparatus embodying thepresent invention;

FIG. 10 is a block diagram of a portion of the apparatus shown in FIG. 9showing details of a mechanical arrangement of a microscope;

FIG. 11 is a flow diagram related to operation of the apparatus;

FIG. 12 is a flow diagram of details of one of the steps in FIG. 11;

FIG. 13 is a display screen showing control parameters to be manipulatedthereon;

FIG. 14 is a flow chart for a region outlying routine;

FIG. 15 is a flow chart for a scanning and analyzing routine;

FIG. 16 is a schematic showing of the limits of travel of the microscopestage with respect to the image tiles;

FIG. 16A is a perspective view of the microscope stage and steppermotors and encoders providing a closed loop drive for the motors;

FIG. 17 is a block diagram of a networked system allowing multipleworkstations to obtain access to the microscope and to manipulate themicroscope locally at each workstation;

FIG. 17A is a view of the system described in connection with FIG. 10;and

FIG. 18 is a block diagram of a remote networked system for distributingand accessing diagnostic images and data, i.e. virtual microscopeslides, through a hypertext transport protocol based server directly orover a packet network.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a block diagram of a system according to the invention forcreating, and transmitting over an intranet or via the Internet avirtual microscope slide, i.e. interrelated data structures and displayprocedures depicting at multiple resolutions, images of a specimen on amicroscope slide. The system includes a microscope with a digitalplatform for supporting the microscope slide. Digital platform or stage11 has been specially calibrated to include a large number of incrementsfor locating portions of specimen images with high precision. Aftercalibration and initial registration of stage 11 in the microscopesetup, a microscope slide or other substrate with a specimen to bescanned is placed on stage 11.

For exemplary purposes, the creation of virtual microscope slidespecimen according to the invention will be described with respect to abreast cancer specimen. The first step in creating a data structureaccording to the invention is to establish a macro image of the entirespecimen (or that portion of the specimen desired to be stored as themacro image). The purpose for creating the macro or large area thumbnailimage is to enable the viewer to see the entire specimen at once and touse the entire image to choose those significant portions thereon forviewing at greater magnification. In this example, the user has selected1.25× as the magnification to display the entire breast cancer slide.Once specimen 13 a has been placed on stage 11, rotating opticalassembly 15 are rotated to select lens 17 which corresponds to the 1.25×magnification.

In accordance with the teachings of the prior patent application, thecomputer controlled microscope is moved to scan the entire image ofspecimen 13 a. The focusing system is programmed to step throughincrements which detect/select only the high resolution center area ofthe field of view in order to avoid storing the fuzzy areas at theperiphery of the field of view. In this example, the macro image will bestored in a 10 by 8 array, for a total of 80 contiguous image tiles, as.shown in FIG. 1A.

A typical microscope slide is about 77 mm by 25 mm, where the usablearea, without including the label, is about 57 mm by 25 m. Each of the80 image segments is about 4.8 mm by 3.5 mm in dimension. This meanseach of the 80 image segments will be scanned separately and stored as aseparate image tile.

The precision of the microscope systems is set up so that each step ofthe motor has a precision of 0.1 micron (micrometer). In this example,the microscope is set up to move 48,143 steps in the X direction and35,800 steps in the Y direction at 1.25× magnification for each of the80 image areas. At higher magnifications, the image areas to scan areconsiderably smaller, so the number of steps is corresponding smaller.For each of the 80 image areas, the microscope lens will detect only thehigh resolution center area of the field of view.

The optical image of the desired image area is then detected by opticalarray sensor 19 (preferably a CCD sensor array). In this example, eachof the 80 scanned areas is sensed by the entire array, which includes752 pixels by 480 pixels. The optical array sensor sends electricalsignals indicative of the detected image to microscope controlledcomputer 32. Computer 32 stores the scanned images, including the topleft X-Y stage coordinates for each of the 80 individual areas of themicroscope slide. Each of the 80 scanned image areas' pixel locationsare stored in a bit mapped file (i.e., a file which contains a map ofthe location of each bit in the area) which corresponds to the layout ofthe individual images thereon. Thus, all of the pixels from the imagetile derived from region A on FIG. 1A, which is the seventh from theleft and in the top row, are individually assigned unique locations inthe computer memory's bit-mapped file (FIG. 6), and are also stored inthe data structure image tile file as shown in FIG. 1B.

Each of the stored data image tiles is a standard image file withextension.bmp, and is of the order of one megabyte, i.e. each of the752×480 pixels is stored as 3 bytes of red, green and blue image data(752×480×32 =1,082,880 bytes). Since the location of each image tile isknown according to the bitmap, the complete microscope image can berecreated by painting (displaying) each image tile in accordance withits grid location. It should be noted

To display the resulting image, computer 32 calculates the appropriateportion to be displayed from each image tile depending upon the relativesize of the display screen. Since the stored image data is usuallygreater than the size of the typical monitor, the viewer must scrollthrough the image on the window to view it entirely. However, anoptional compression algorithm can be used to compress the entire imageinto the viewing window. The X-Y coordinate information is used by theviewing and manipulation program to reconstruct the image tiles into acomplete image of the specimen. The resulting image is larger, and withbetter resolution than would be achieved if optics technology were ableto construct a single lens capable of viewing the entire specimen in onefield of view. In this example, each of the 80 image tiles has digitalresolution of 752×480 pixels, with corresponding optical resolution ofapproximate 0.2 microns at 40× to approximately 6.4 microns at 1.25×.

After the macro or thumbnail images are digitally scanned and storedwith their X-Y coordinate information, the user then examines the macroimage or original specimen for significant details. Typically, the userwill highlight with a marking pen the areas to be viewed at highermagnification. The user then changes the magnification of optics' system15 to the desired higher magnification, moves the scanning system tobring the selected region into view. Computer 32 then repeats thescanning and image tile creation process for the selected region, but athigher magnification and with a new grid system to locate the scannedselected regions.

In the example, the user has selected region B shown on FIG. 1A toperform a second view at a higher magnification. The user selects, forexample, a 40× magnification. The computer calculates the number oftiles to cover the selected area at 40× magnification and sets up asecond grid.

It should be noted that region B crosses over several of the largertiles in FIG. 1A. Because of the extreme precision of the instrument,0.1 micron resolution, locating such selected regions with highresolution is readily accomplished. As noted above, the computecalculates the size of the image portion, in this case as an example,X=1500 and Y=1200 stepping increments. Each image portion at the 40×resolution is detected by the optical sensor array, 752 by 480 pixels.Each resulting data file is stored in a separate, high magnificationmapped area of memory so that the computer can easily recall thelocation of region B, or any of its 200 individual image tiles, whenrequested by a user.

Once the user has completed selecting and having the computer controlledmicroscope system scan and store the digital images in image tiles,computer 32 stores the mapped .bmp files along with their coordinateinformation and creates slide image data structure 31 in FIG. 1. Slideimage data structure includes all of the bitmap image tile files at bothmagnifications (note that similarly, additional images could be storedat further magnifications, if desired), as well as X-Y coordinateinformation for the location of the various image tiles.

FIG. 7A is a file listing such as would be seen under a Windows 95 filemanager showing the data files included in a data structure for a breastcancer specimen. Included in the file listing are FinalScan.ini andSlideScan.ini as well as sixty bitmap data files. Slidescan.ini is alisting of all the original bitmap (.bmp) files. The bitmap filesrepresent the individual image tiles in the scan at, say, 1.25×magnifications. Slidescan.ini is set forth below in Table 1 anddescribes the X-Y coordinates for each image tile file. When the datastructure is viewed by a control program, the program uses the X-Ycoordinates to display all the image tiles contiguously.

TABLE 1 Slidescan.ini [Header] x = 278000 y = 142500 lXStepSize = 48143lYStepSize = 35800 iScannedCount = 37 [Ss1] x = 181714 y = 142500 [Ss2]x = 133571 y = 142500 [Ss3] x = 37285 y = 106700 [Ss4] x = 85428 y =106700 [Ss5] x = 133571 y = 106700 [Ss6] x = 181714 y = 106700 [Ss7] x =229857 y = 106700 [Ss8] x = 229857 y = 70900 [Ss9] x = 181714 y = 70900[Ss10] x = 133571 y = 70900 [Ss11] x = 85428 y = 70900 [Ss12] x = 37285y = 70900 [Ss13] x = −10858 y = 70900 [Ss14] x = −10858 y = 35100 [Ss15]x = 37285 y = 35100 [Ss16] x = 85428 y = 35100 [Ss17] x = 133571 y =35100 [Ss18] x = 181714 y = 35100 [Ss19] x = 229857 y = 35100 [Ss20] x =278000 y = −700 [Ss21] x = 229857 y = −700 [Ss22] x = 181714 y = −700[Ss23] x = 133571 y = −700 [Ss24] x = 85428 y = −700 [Ss25] x = 37285 y= −700 [Ss26] x = −10858 y = −700 [Ss27] x = −10858 y = −36500 [Ss28] x= 37285 y = −36500 [Ss29] x = 85428 y = −36500 [Ss30] x = 133571 y =−36500 [Ss31] x = 181714 y = −36500 [Ss32] x = 229857 y = −36500 [Ss33]x = 278000 y = −36500 [Ss34] x = 278000 y = −72300 [Ss35] x = 229857 y =−72300 [Ss36] x = 181714 y = −72300 [Ss37] x = 133571 y = −72300Table 2 is a listing of the file FinalScan.ini, which is a listing theX-Y coordinates of the high magnification image tiles scanned andstored.

TABLE 2 FinalScan.ini [Header] tPatientID = mDa027 tAccession =tOperatorID = jwb tTimeOfScan = 8/4/97 1:19:56 PM lXStageRef = 278000lYStageRef = 142500 iImageWidth = 752 iImageHeight = 480 lXStepSize =1590 lYStepSize = 1190 lXOffset = −1900 lYOffset = −400 dMagnification =40 lAnalysisImageCount = 105 lCalibrationImageCount = 0 [Da0] x = 214532y = 65584 [Da1] x = 212996 y = 65584 [Da2] x = 211460 y = 65584 [Da3] x= 209924 y = 65584 [Da4] x = 208388 y = 65584 [Da5] x = 206852 y = 65584[Da6] x = 205316 y = 65584 [Da7] x = 203780 y = 65584 [Da8] x = 214532 y= 64400 [Da9] x = 212996 y = 64400 [Da10] x = 211460 y = 64400 [Da11] x= 209924 y = 64400 [Da12] x = 208388 y = 64400 [Da13] x = 206852 y =64400 [Da14] x = 205316 y = 64400 [Da15] x = 203780 y = 64400 [Da16] x =214532 y = 63216 [Da17] x = 212996 y = 63216 [Da18] x = 211460 y = 63216[Da19] x = 209924 y = 63216 [Da20] x = 208388 y = 63216 [Da21] x =206852 y = 63216 [Da22] x = 205316 y = 63216 [Da23] x = 203780 y = 63216[Da24] x = 214532 y = 62032 [Da25] x = 212996 y = 62032 [Da26] x =211460 y = 62032 [Da27] x = 209924 y = 62032 [Da28] x = 208388 y = 62032[Da29] x = 206852 y = 62032 [Da30] x = 205316 y = 62032 [Da31] x =203780 y = 62032 [Da32] x = 214532 y = 60848 [Da33] x = 212996 y = 60848[Da34] x = 211460 y = 60848 [Da35] x = 209924 y = 60848 [Da36] x =208388 y = 60848 [Da37] x = 206852 y = 60848 [Da38] x = 205316 y = 60848[Da39] x = 203780 y = 60848 [Da40] x = 214532 y = 59664 [Da41] x =212996 y = 59664 [Da42] x = 211460 y = 59664 [Da43] x = 209924 y = 59664[Da44] x = 208388 y = 59664 [Da45] x = 206852 y = 59664 [Da46] x =205316 y = 59664 [Da47] x = 203780 y = 59664 [Da48] x = 214532 y = 58480[Da49] x = 212996 y = 58480 [Da50] x = 211460 y = 58480 [Da51] x =209924 y = 58480 [Da52] x = 208388 y = 58480 [Da53] x = 206852 y = 58480[Da54] x = 205316 y = 58480 [Da55] x = 203780 y = 58480 [Da56] x =180740 y = 82160 [Da57] x = 179204 y = 82160 [Da58] x = 177668 y = 82160[Da59] x = 176132 y = 82160 [Da60] x = 174596 y = 82160 [Da61] x =173060 y = 82160 [Da62] x = 171524 y = 82160 [Da63] x = 180740 y = 80976[Da64] x = 179204 y = 80976 [Da65] x = 177668 y = 80976 [Da66] x =176132 y = 80976 [Da67] x = 174596 y = 80976 [Da68] x = 173060 y = 80976[Da69] x = 171524 y = 80976 [Da70] x = 180740 y = 79792 [Da71] x =179204 y = 79792 [Da72] x = 177668 y = 79792 [Da73] x = 176132 y = 79792[Da74] x = 174596 y = 79792 [Da75] x = 173060 y = 79792 [Da76] x =171524 y = 79792 [Da77] x = 180740 y = 78608 [Da78] x = 179204 y = 78608[Da79] x = 177668 y = 78608 [Da80] x = 176132 y = 78608 [Da81] x =174596 y = 78608 [Da82] x = 173060 y = 78608 [Da83] x = 171524 y = 78608[Da84] x = 180740 y = 77424 [Da85] x = 179204 y = 77424 [Da86] x =177668 y = 77424 [Da87] x = 176132 y = 77424 [Da88] x = 174596 y = 77424[Da89] x = 173060 y = 77424 [Da90] x = 171524 y = 77424 [Da91] x =180740 y = 76240 [Da92] x = 179204 y = 76240 [Da93] x = 177668 y = 76240[Da94] x = 176132 y = 76240 [Da95] x = 174596 y = 76240 [Da96] x =173060 y = 76240 [Da97] x = 171524 y = 76240 [Da98] x = 180740 y = 75056[Da99] x = 179204 y = 75056 [Da100] x = 177668 [Da101] x = 176132 y =75056 [Da102] x = 174596 y = 75056 [Da103] x = 173060 y = 75056 [Da104]x = 171524 y = 75056

Computer 32 can also use the scanned image files to create aself-executing data structure. By compressing the .bmp images to .jpgand adding a dynamic, self-executing program which enables the user toview, reconstruct and manipulate the image tiles, the user can use thedata structure as a virtual microscope slide of the original specimen.Preferably, the dynamic, self-executing program is a Java applet, suchas shown on FIG. 7B.

Computer 32 can provide the slide image data structure 31 directly orvia an intranet browser 33 to local viewer 34, or via an Internet server38. Slide image data structure 37 is shown as being directly accessiblefrom Internet server 38. Alternatively, a user can download the slideimage data structure on his own computer 39, use an internet browser 43and view the reconstructed images. Another alternative is for computer32 to store the slide image data structure on a CD-rom, Jazz drive orother storage medium.

To view slide image data structure 31 or 37, the user, who for example,has acquired the data structure via a CD-rom, first installs the CD-romin the CD-rom drive of his computer. Then the user opens up a browser orother applications program which can read the Java applet installed onthe CD-rom with the image tiles. Note that in some instances no separatebrowser program may be required. In some case, the CD-rom may includethe complete applications program for viewing, reconstructing andmanipulating the image tiles. In the instant example, the user will thenselect the icon or file listing for the slide image data structure andthe control program will display the data files.

FIG. 2 is a screen view of a system embodying the present inventionshowing a low magnification image 24 of a specimen on a microscope slidein one window, a high magnification image 26 of a portion of the lowmagnification image selected by a region marker 30 and a control window28. FIG. 3 is a view of a display screen of the apparatus embodying thepresent invention showing the control window 28, a low magnificationwindow 24 having a plurality of high magnification micro image regions310 delineated therein and a high magnification window 26 including oneor more of the micro image regions 310, 314, 316. FIG. 4 is a view of amacro image of an actual breast cancer specimen displayed at 1.25× asseen on a computer monitor. FIG. 5 is a view of the grid portion of FIG.4 outlining a region of interest selected by a pathologist displayed at40× magnification.

Recall that region A in FIG. 1A was about 4.8 mm by 3.5 mm. This areacreates 752 by 480 pixels of sensed data, or 360,930 pixels ofinformation. Each pixel sends information about its location and theimage it sensed to the computer. The computer stores this information ina series of data files (typically .bmp format, but .tif or .gif couldalso be used). Thus, it can be seen that several more pixels of senseddata are available for viewing on a computer monitor operating at 640 by480. To view the entire image, the user must scroll through the imagetiles. However, scrolling need not be done on a tile, by tile basis.Rather, the user scrolls by pointing to a pixel on the monitor.

FIG. 6 is a block diagram showing how the control program locates andscrolls through the stored image tiles. Using the example from FIG. 1 a,a complete data structure has been created. When the user loads the datastructure (of the microscope slide) into his personal computer or viewsit from an Internet browser, the control program recreates a bit map ofthe stored data. The bit map of the entire slide is shown in FIG. 6.Image tile A is also high-lighted. This bit map enables a user to pointto or otherwise reference a location on the slide.

The X-Y coordinate information'specified in the data structure enablesX-Y translation of the specific image tiles and specific pixels withinthe image tile. When the control program first loads the image, becausethis image file is so large, only a small number of the available tilesare displayed in the active window on the user's monitor. The user useshis mouse or pointing device to scroll through the active window to viewthe entire macro image. The X-Y coordinate information selected by themouse translates into specific image. tiles or portions therein. Thecomputer takes the mouse pointer information and retrieves the imagedata from the series of stored tile images and displays them on themonitor for viewing the by user.

Because of the large amount of CCD pixel information stored, actual CCDpixel information can be recreated in the viewing window. The entiresystem operates in a loop, where the user inputs a mouse location, thecomputer translates the mouse location from the screen coordinates(screen pixels) to the X-Y coordinates on the bit map.

Similarly, the user may select the high magnification data images. Theseare outlined by a dark grid, indicating the areas stored. The useroperates the mouse in the same manner as described above. The controlprogram locates the stored X-Y coordinates and retrieves the selectedparts of the image, CCD stored pixel by CCD stored pixel.

As mentioned above, to save storage space, computer 32 can perform adata compression on each of the image tile files. A preferred datacompression is JPEG, which is readily transferred and recognized by mostInternet browser programs. Also, JPEG allows flexibility in the amountof data to be compressed, from 20 to 80 percent. FIG. 8 is file listingsuch as would be seen under Windows 95 file manager showing the datafiles included in an alternate data structure, one in which the datafiles have been compressed or converted to JPEG (.jpg) format for abreast cancer specimen. The file index.html (shown in Table 3) is thelisting which contains the X-Y coordinate information for these datafiles. This is the information that is read by the dynamic,self-executing program for viewing, reconstructing and manipulating theimage tiles into the macro and micro views.

TABLE 3 index.html <HTML> <TITLE> DCIS_027 - Web Slide </TITLE> <BODY><APPLET CODE=WebSlide/BliWebSlide.class NAME=DCIS_027 WIDTH=3384HEIGHT=960 HSPACE=0 VSPACE=0 ALIGN=Middle> <PARAM NAME = “tPatientID”VALUE = “mda027”> <PARAM NAME = “tAccession” VALUE = “”> <PARAM NAME =“tOperatorID” VALUE = “jwb”> <PARAM NAME = “tTimeOfScan” VALUE = “8/4/971:19:56 PM”> <PARAM NAME = “lXStageRef” VALUE = “278000”> <PARAM NAME =“lYStageRef” VALUE = “142500”> <PARAM NAME = “iImageWidth” VALUE =“752”> <PARAM NAME = “iImageHeight” VALUE = “480”> <PARAM NAME =“lXStepSize” VALUE = “1590”> <PARAM NAME = “lYStepSize” VALUE = “1190”><PARAM NAME = “lXOffset” VALUE = “−1900”> <PARAM NAME = “lYOffset” VALUE= “−400”> <PARAM NAME = “dMagnification” VALUE = “40”> <PARAM NAME =“iImageCount” VALUE = “105”> <PARAM NAME = “lXSsStepSize” VALUE =“48143”> <PARAM NAME = “lYSsStepSize” VALUE = “35800”> <PARAM NAME =“iScannedCount” VALUE = “37”> <PARAM NAME = “lStartX” VALUE = “278000”><PARAM NAME = “lStartY” VALUE = “142500”> <PARAM NAME = “Ss1_X” VALUE =“181714”> <PARAM NAME = “Ss1_Y” VALUE = “142500”> <PARAM NAME = “Ss2_X”VALUE = “133571”> <PARAM NAME = “Ss2_Y” VALUE = “142500”> <PARAM NAME =“Ss3_X” VALUE = “37285”> <PARAM NAME = “Ss3_Y” VALUE = “106700”> <PARAMNAME = “Ss4_X” VALUE = “85428”> <PARAM NAME = “Ss4_Y” VALUE = “106700”><PARAM NAME = “Ss5_X” VALUE = “133571”> <PARAM NAME = “Ss5_Y” VALUE =“106700”> <PARAM NAME = “Ss6_X” VALUE = “181714”> <PARAM NAME = “Ss6_Y”VALUE = “106700”> <PARAM NAME = “Ss7_X” VALUE = “229857”> <PARAM NAME =“Ss7_Y” VALUE = “106700”> <PARAM NAME = “Ss8_X” VALUE = “229857”> <PARAMNAME = “Ss8_Y” VALUE = “70900”> <PARAM NAME = “Ss9_X” VALUE = “181714”><PARAM NAME = “Ss9_Y” VALUE = “70900”> <PARAM NAME = “Ss10_X” VALUE =“133571”> <PARAM NAME = “Ss10_Y” VALUE = “70900”> <PARAM NAME = “Ss11_X”VALUE = “85428”> <PARAM NAME = “Ss11_Y” VALUE = “70900”> <PARAM NAME =“Ss12_X” VALUE = “37285”> <PARAM NAME = “Ss12_Y” VALUE = “70900”> <PARAMNAME = “Ss13_X” VALUE = “−10858”> <PARAM NAME = “Ss13_Y” VALUE =“70900”> <PARAM NAME = “Ss14_X” VALUE = “−10858”> <PARAM NAME = “Ss14_Y”VALUE = “35100”> <PARAM NAME = “Ss15_X” VALUE = “37285”> <PARAM NAME =“Ss15_Y” VALUE = “35100”> <PARAM NAME = “Ss16_X” VALUE = “85428”> <PARAMNAME = “Ss16_Y” VALUE = “35100”> <PARAM NAME = “Ss17_X” VALUE =“133571”> <PARAM NAME = “Ss17_Y” VALUE = “35100”> <PARAM NAME = “Ss18_X”VALUE = “181714”> <PARAM NAME = “Ss18_Y” VALUE = “35100”> <PARAM NAME =“Ss19_X” VALUE = “229857”> <PARAM NAME = “Ss19_Y” VALUE = “35100”><PARAM NAME = “Ss20_X” VALUE = “278000”> <PARAM NAME = “Ss20_Y” VALUE =“−700”> <PARAM NAME = “Ss21_X” VALUE = “229857”> <PARAM NAME = “Ss21_Y”VALUE = “−700”> <PARAM NAME = “Ss22_X” VALUE = “181714”> <PARAM NAME =“Ss22_Y” VALUE = “−700”> <PARAM NAME = “Ss23_X” VALUE = “133571”> <PARAMNAME = “Ss23_Y” VALUE = “−700”> <PARAM NAME = “Ss24_X” VALUE = “85428”><PARAM NAME = “Ss24_Y” VALUE = “−700”> <PARAM NAME = “Ss25_X” VALUE =“37285”> <PARAM NAME = “Ss25_Y” VALUE = “−700”> <PARAM NAME = “Ss26_X”VALUE = “−10858”> <PARAM NAME = “Ss26_Y” VALUE = “−700”> <PARAM NAME =“Ss27_X” VALUE = “−10858”> <PARAM NAME = “Ss27_Y” VALUE = “−36500”><PARAM NAME = “Ss28_X” VALUE = “37285”> <PARAM NAME = “Ss28_Y” VALUE =“−36500”> <PARAM NAME = “Ss29_X” VALUE = “85428”> <PARAM NAME = “Ss29_Y”VALUE = “−36500”> <PARAM NAME = “Ss30_X” VALUE = “133571”> <PARAM NAME =“Ss30_Y” VALUE = “−36500”> <PARAM NAME = “Ss31_X” VALUE = “181714”><PARAM NAME = “Ss31_Y” VALUE = “−36500”> <PARAM NAME = “Ss32_X” VALUE =“229857”> <PARAM NAME = “Ss32_Y” VALUE = “−36500”> <PARAM NAME =“Ss33_X” VALUE = “278000”> <PARAM NAME = “Ss33_Y” VALUE = “−36500”><PARAM NAME = “Ss34_X” VALUE = “278000”> <PARAM NAME = “Ss34_Y” VALUE =“−72300”> <PARAM NAME = “Ss35_X” VALUE = “229857”> <PARAM NAME =“Ss35_Y” VALUE = “−72300”> <PARAM NAME = “Ss36_X” VALUE = “181714”><PARAM NAME = “Ss36_Y” VALUE = “−72300”> <PARAM NAME = “Ss37_X” VALUE =“133571”> <PARAM NAME = “Ss37_Y” VALUE = “−72300”> <PARAM NAME = “Da0_X”VALUE = “214532”> <PARAM NAME = “Da0_Y” VALUE = “65584”> <PARAM NAME =“Da1_X” VALUE = “212996”> <PARAM NAME = “Da1_Y” VALUE = “65584”> <PARAMNAME = “Da2_X” VALUE = “211460”> <PARAM NAME = “Da2_Y” VALUE = “65584”><PARAM NAME = “Da3_X” VALUE = “209924”> <PARAM NAME = “Da3_Y” VALUE =“65584”> <PARAM NAME = “Da4_X” VALUE = “208388”> <PARAM NAME = “Da4_Y”VALUE = “65584”> <PARAM NAME = “Da5_X” VALUE = “206852”> <PARAM NAME =“Da5_Y” VALUE = “65584”> <PARAM NAME = “Da6_X” VALUE = “205316”> <PARAMNAME = “Da6_Y” VALUE = “65584”> <PARAM NAME = “Da7_X” VALUE = “203780”><PARAM NAME = “Da7_Y” VALUE = “65584”> <PARAM NAME = “Da8_X” VALUE =“214532”> <PARAM NAME = “Da8_Y” VALUE = “64400”> <PARAM NAME = “Da9_X”VALUE = “212996”> <PARAM NAME = “Da9_Y” VALUE = “64400”> <PARAM NAME =“Da10_X” VALUE = “211460”> <PARAM NAME = “Da10_Y” VALUE = “64400”><PARAM NAME = “Da11_X” VALUE = “209924”> <PARAM NAME = “Da11_Y” VALUE =“64400”> <PARAM NAME = “Da12_X” VALUE = “208388”> <PARAM NAME = “Da12_Y”VALUE = “64400”> <PARAM NAME = “Da13_X” VALUE = “206852”> <PARAM NAME =“Da13_Y” VALUE = “64400”> <PARAM NAME = “Da14_X” VALUE = “205316”><PARAM NAME = “Da14_Y” VALUE = “64400”> <PARAM NAME = “Da15_X” VALUE =“203780”> <PARAM NAME = “Da15_Y” VALUE = “64400”> <PARAM NAME = “Da16_X”VALUE = “214532”> <PARAM NAME = “Da16_Y” VALUE = “63216”> <PARAM NAME =“Da17_X” VALUE = “212996”> <PARAM NAME = “Da17_Y” VALUE = “63216”><PARAM NAME = “Da18_X” VALUE = “211460”> <PARAM NAME = “Da18_Y” VALUE =“63216”> <PARAM NAME = “Da19_X” VALUE = “209924”> <PARAM NAME = “Da19_Y”VALUE = “63216”> <PARAM NAME = “Da20_X” VALUE = “208388”> <PARAM NAME =“Da20_Y” VALUE = “63216”> <PARAM NAME = “Da21_X” VALUE = “206852”><PARAM NAME = “Da21_Y” VALUE = “63216”> <PARAM NAME = “Da22_X” VALUE =“205316”> <PARAM NAME = “Da22_Y” VALUE = “63216”> <PARAM NAME = “Da23_X”VALUE = “203780”> <PARAM NAME = “Da23_Y” VALUE = “63216”> <PARAM NAME =“Da24_X” VALUE = “214532”> <PARAM NAME = “Da24_Y” VALUE = “62032”><PARAM NAME = “Da25_X” VALUE = “212996”> <PARAM NAME = “Da25_Y” VALUE =“62032”> <PARAM NAME = “Da26_X” VALUE = “211460”> <PARAM NAME = “Da26_Y”VALUE = “62032”> <PARAM NAME = “Da27_X” VALUE = “209924”> <PARAM NAME =“Da27_Y” VALUE = “62032”> <PARAM NAME = “Da28_X” VALUE = “208388”><PARAM NAME = “Da28_Y” VALUE = “62032”> <PARAM NAME = “Da29_X” VALUE =“206852”> <PARAM NAME = “Da29_Y” VALUE = “62032”> <PARAM NAME = “Da30_X”VALUE = “205316”> <PARAM NAME = “Da30_Y” VALUE = “62032”> <PARAM NAME =“Da31_X” VALUE = “203780”> <PARAM NAME = “Da31_Y” VALUE = “62032”><PARAM NAME = “Da32_X” VALUE = “214532”> <PARAM NAME = “Da32_Y” VALUE =“60848”> <PARAM NAME = “Da33_X” VALUE = “212996”> <PARAM NAME = “Da33_Y”VALUE = “60848”> <PARAM NAME = “Da34_X” VALUE = “211460”> <PARAM NAME =“Da34_Y” VALUE = “60848”> <PARAM NAME = “Da35_X” VALUE = “209924”><PARAM NAME = “Da35_Y” VALUE = “60848”> <PARAM NAME = “Da36_X” VALUE =“208388”> <PARAM NAME = “Da36_Y” VALUE = “60848”> <PARAM NAME = “Da37_X”VALUE = “206852”> <PARAM NAME = “Da37_Y” VALUE = “60848”> <PARAM NAME =“Da38_X” VALUE = “205316”> <PARAM NAME = “Da38_Y” VALUE = “60848”><PARAM NAME = “Da39_X” VALUE = “203780”> <PARAM NAME = “Da39_Y” VALUE =“60848”> <PARAM NAME = “Da40_X” VALUE = “214532”> <PARAM NAME = “Da40_Y”VALUE = “59664”> <PARAM NAME = “Da41_X” VALUE = “212996”> <PARAM NAME =“Da41_Y” VALUE = “59664”> <PARAM NAME = “Da42_X” VALUE = “211460”><PARAM NAME = “Da42_Y” VALUE = “59664”> <PARAM NAME = “Da43_X” VALUE =“209924”> <PARAM NAME = “Da43_Y” VALUE = “59664”> <PARAM NAME = “Da44_X”VALUE = “208388”> <PARAM NAME = “Da44_Y” VALUE = “59664”> <PARAM NAME =“Da45_X” VALUE = “206852”> <PARAM NAME = “Da45_Y” VALUE = “59664”><PARAM NAME = “Da46_X” VALUE = “205316”> <PARAM NAME = “Da46_Y” VALUE =“59664”> <PARAM NAME = “Da47_X” VALUE = “203780”> <PARAM NAME = “Da47_Y”VALUE = “59664”> <PARAM NAME = “Da48_X” VALUE = “214532”> <PARAM NAME =“Da48_Y” VALUE = “58480”> <PARAM NAME = “Da49_X” VALUE = “212996”><PARAM NAME = “Da49_Y” VALUE = “58480”> <PARAM NAME = “Da50_X” VALUE =“211460”> <PARAM NAME = “Da50_Y” VALUE = “58480”> <PARAM NAME = “Da51_X”VALUE = “209924”> <PARAM NAME = “Da51_Y” VALUE = “58480”> <PARAM NAME =“Da52_X” VALUE = “208388”> <PARAM NAME = “Da52_Y” VALUE = “58480”><PARAM NAME = “Da53_X” VALUE = “206852”> <PARAM NAME = “Da53_Y” VALUE =“58480”> <PARAM NAME = “Da54_X” VALUE = “205316”> <PARAM NAME = “Da54_Y”VALUE = “58480”> <PARAM NAME = “Da55_X” VALUE = “203780”> <PARAM NAME =“Da55_Y” VALUE = “58480”> <PARAM NAME = “Da56_X” VALUE = “180740”><PARAM NAME = “Da56_Y” VALUE = “82160”> <PARAM NAME = “Da57_X” VALUE =“179204”> <PARAM NAME = “Da57_Y” VALUE = “82160”> <PARAM NAME = “Da58_X”VALUE = “177668”> <PARAM NAME = “Da58_Y” VALUE = “82160”> <PARAM NAME =“Da59_X” VALUE = “176132”> <PARAM NAME = “Da59_Y” VALUE = “82160”><PARAM NAME = “Da60_X” VALUE = “174596”> <PARAM NAME = “Da60_Y” VALUE =“82160”> <PARAM NAME = “Da61_X” VALUE = “173060”> <PARAM NAME = “Da61_Y”VALUE = “82160”> <PARAM NAME = “Da62_X” VALUE = “171524”> <PARAM NAME =“Da62_Y” VALUE = “82160”> <PARAM NAME = “Da63_X” VALUE = “180740”><PARAM NAME = “Da63_Y” VALUE = “80976”> <PARAM NAME = “Da64_X” VALUE =“179204”> <PARAM NAME = “Da64_Y” VALUE = “80976”> <PARAM NAME = “Da65_X”VALUE = “177668”> <PARAM NAME = “Da65_Y” VALUE = “80976”> <PARAM NAME =“Da66_X” VALUE = “176132”> <PARAM NAME = “Da66_Y” VALUE = “80976”><PARAM NAME = “Da67_X” VALUE = “174596”> <PARAM NAME = “Da67_Y” VALUE =“80976”> <PARAM NAME = “Da68_X” VALUE = “173060”> <PARAM NAME = “Da68_Y”VALUE = “80976”> <PARAM NAME = “Da69_X” VALUE = “171524”> <PARAM NAME =“Da69_Y” VALUE = “80976”> <PARAM NAME = “Da70_X” VALUE = “180740”><PARAM NAME = “Da70_Y” VALUE = “79792”> <PARAM NAME = “Da71_X” VALUE =“179204”> <PARAM NAME = “Da71_Y” VALUE = “79792”> <PARAM NAME = “Da72_X”VALUE = “177668”> <PARAM NAME = “Da72_Y” VALUE = “79792”> <PARAM NAME =“Da73_X” VALUE = “176132”> <PARAM NAME = “Da73_Y” VALUE = “79792”><PARAM NAME = “Da74_X” VALUE = “174596”> <PARAM NAME = “Da74_Y” VALUE =“79792”> <PARAM NAME = “Da75_X” VALUE = “173060”> <PARAM NAME = “Da75_Y”VALUE = “79792”> <PARAM NAME = “Da76_X” VALUE = “171524”> <PARAM NAME =“Da76_Y” VALUE = “79792”> <PARAM NAME = “Da77_X” VALUE = “180740”><PARAM NAME = “Da77_Y” VALUE = “78608”> <PARAM NAME = “Da78_X” VALUE =“179204”> <PARAM NAME = “Da78_Y” VALUE = “78608”> <PARAM NAME = “Da79_X”VALUE = “177668”> <PARAM NAME = “Da79_Y” VALUE = “78608”> <PARAM NAME =“Da80_X” VALUE = “176132”> <PARAM NAME = “Da80_Y” VALUE = “78608”><PARAM NAME = “Da81_X” VALUE = “174596”> <PARAM NAME = “Da81_Y” VALUE =“78608”> <PARAM NAME = “Da82_X” VALUE = “173060”> <PARAM NAME = “Da82_Y”VALUE = “78608”> <PARAM NAME = “Da83_X” VALUE = “171524”> <PARAM NAME =“Da83_Y” VALUE = “78608”> <PARAM NAME = “Da84_X” VALUE = “180740”><PARAM NAME = “Da84_Y” VALUE = “77424”> <PARAM NAME = “Da85_X” VALUE =“179204”> <PARAM NAME = “Da85_Y” VALUE = “77424”> <PARAM NAME = “Da86_X”VALUE = “177668”> <PARAM NAME = “Da86_Y” VALUE = “77424”> <PARAM NAME =“Da87_X” VALUE = “176132”> <PARAM NAME = “Da87_Y” VALUE = “77424”><PARAM NAME = “Da88_X” VALUE = “174596”> <PARAM NAME = “Da88_Y” VALUE =“77424”> <PARAM NAME = “Da89_X” VALUE = “173060”> <PARAM NAME = “Da89_Y”VALUE = “77424”> <PARAM NAME = “Da90_X” VALUE = “171524”> <PARAM NAME =“Da90_Y” VALUE = “77424”> <PARAM NAME = “Da91_X” VALUE = “180740”><PARAM NAME = “Da91_Y” VALUE = “76240”> <PARAM NAME = “Da92_X” VALUE =“179204”> <PARAM NAME = “Da92_Y” VALUE = “76240”> <PARAM NAME = “Da93_X”VALUE = “177668”> <PARAM NAME = “Da93_Y” VALUE = “76240”> <PARAM NAME =“Da94_X” VALUE = “176132”> <PARAM NAME = “Da94_Y” VALUE = “76240”><PARAM NAME = “Da95_X” VALUE = “174596”> <PARAM NAME = “Da95_Y” VALUE =“76240”> <PARAM NAME = “Da96_X” VALUE = “173060”> <PARAM NAME = “Da96_Y”VALUE = “76240”> <PARAM NAME = “Da97_X” VALUE = “171524”> <PARAM NAME =“Da97_Y” VALUE = “76240”> <PARAM NAME = “Da98_X” VALUE = “180740”><PARAM NAME = “Da98_Y” VALUE = “75056”> <PARAM NAME = “Da99_X” VALUE =“179204”> <PARAM NAME = “Da99_Y” VALUE = “75056”> <PARAM NAME =“Da100_X” VALUE = “177668”> <PARAM NAME = “Da100_Y” VALUE = “75056”><PARAM NAME = “Da101_X” VALUE = “176132”> <PARAM NAME = “Da101_Y” VALUE= “75056”> <PARAM NAME = “Da102_X” VALUE = “174596”> <PARAM NAME =“Da102_Y” VALUE = “75056”> <PARAM NAME = “Da103_X” VALUE = “173060”><PARAM NAME = “Da103_Y” VALUE = “75056”> <PARAM NAME = “Da104_X” VALUE =“171524”> <PARAM NAME = “Da104_Y” VALUE = “75056”> </APPLET> </BODY></HTML>

Referring now to the drawings, and especially to FIGS. 9A, 9B and 10,apparatus for synthesizing low magnification and high magnificationmicroscopic images is shown therein and generally identified byreference numeral 10. The system includes a computer 12 which is a dualPentium Pro personal computer in combination with a Hitachi HV-C20 videocamera 14 associated with a Zeiss Axioplan 2 microscope 16. The computersystem 12 is able to receive signals from the camera 14 which captureslight from the microscope 16 having a microscope slide 18 positioned onan LUDL encoded motorized stage 20. The encoded motorized stage 20;includes a MAC 2000 stage controller for controlling the stage inresponse to the computer 12. A microscope slide 18 includes a biologicalspecimen 21 which is to be viewed by the microscope and whose image isto be digitized both at low magnification and at high magnification asselected by a user. The low magnification digitized image is thendisplayed on a 21 inch Iiyama video display monitor 22 having resolutionof 1600 by 1200 to provide display screens of the type shown in FIGS. 1through 3 including a low magnification image 24, for instance, at 1.25power, a high magnification image 26, for instance at 40× power and acontrol window or image 28. The low magnification image may haveidentified therein a region 30 which is reproduced at high magnificationin high magnification screen or window 26 so that a pathologist or otheroperator of the system can review architectural regions of interest inlow magnification image 24 and simultaneously view them in highmagnification in the high magnification screen or window 26 to determinewhether the cells forming a portion of the architectural feature need beexamined further for cancer or the like or not.

The computer 10 is constructed around a PCI system bus 40 and has afirst Pentium Pro microprocessor 42 and a second pentium promicroprocessor 44 connected thereto. The system bus 40 has connected toit a PCI bus 50 and an ISA bus 52. The PCI bus 50 has a SCSI controller60 connected thereto to send and receive information from a hard disk62. The hard disk 62 also is coupled in daisy chain SCSI fashion to ahigh capacity removal disk and to a CD Rom drive 66. The hard disks 62contains the programs for operating the system for controlling themicroscope 16 and for processing the images as well as for doing aquantitative analysis of the selected portions of the histologicalspecimens being viewed on the slide 18. The system bus 40 also hasconnected to it a random access memory 70 within which portions of theprogram being executed are stored as well as a read only memory 72 forholding a bootstrap loader as well as portions of the basic input/outputoperating system. A floppy disk controller 74 is coupled to the systembus 40 and has connected to it a floppy disk drive 76 for reading andwriting information to a floppy disk as appropriate. A mouse controller80 is coupled to the system bus and has a mouse 82 which operates as apointing device for controlling manipulations on the screen 22 andwithin the windows 24, 26 and 28. A keyboard controller 90 is connectedto the system bus and has a keyboard 92 connected thereto. The keyboard92 may be used to send and receive alpha numeric signals to otherportions of the computer. An audio controller 100 has a plurality ofspeakers 102 and a microphone 104 connected thereto for audio input andoutput and is coupled to the system bus 40. A network interface, such asa network interface card 104, is connected to the system bus and canprovide signals via a channel 106 to other portions of a network orinternet to which the system may be connected. Likewise, signals can besent out of the system through a modem 110 connected to the ISA bus 52and may be sent via a channel 112, for instance, to the internet. Aprinter 116 is connected via a parallel I/O controller 118 to the systembus in order to provide printouts as appropriate of screens and otherinformation as it is generated. A serial I/O controller 122 is connectedto the system bus and has connected to it a camera controller 124 whichis coupled to CCD sensors 126 in the cameras. The CCD sensors 126 supplypixel or image signals representative of what is found on the slide 18to an Epix pixci image acquisition controller 130 coupled to the PCI bus50.

The microscope 16 includes a base 140 having a stage 20 positionedthereon as well as an objective turret 142 having a plurality ofobjectives 144, 146 and 148 thereon. The objective 144, for instance,may be of 1.25× objective. The objective 146 may be a 20× objective. Theobjective 148 may be a 40× objective. Signals from the camera sensorsand controller are supplied over a bus 128 to the image acquisitionsystem where they are digitized and supplied to the PCI bus for storagein RAM or for backing storage on the hard disk 62.

When a specimen is on the slide 18 the stage 20 may be manipulated underthe control of the computer through a stage controller 160 coupled tothe serial I/O controller 122. Likewise, a microscope controller 162controls aspects of the microscope such as the illumination, the colortemperature or spectral output of a lamp 168 and the like. For instance,in normal operation, when a specimen is placed on the slide, specimenslide 18 is placed on the stage 20 in a step 200, as shown in FIG. 14,the processors 42 or 44 send a command through the system bus to causethe serial I/O controller 122 to signal the microscope controller tochange magnification to 1.25× in a step 202. This is done by rotatingthe objective turret of the Axioplan 2 microscope to select theobjective 144. Likewise, the controller sets the color temperature ofthe lamp 168, sets a pair of neutral density filter wheels 170 and 172and sets a field diaphragm 174 for the correct illumination. A condenserdiaphragm 176 is also controlled and a color filter wheel 180 may alsobe controlled to apply the appropriate filter color to the CCD censors126 in the camera. The entire slide is then scanned in a step 204. Theimages are tiled and melded together into the overall image 24 suppliedon the screen 22 to provide the operator in the step 206 with a visuallyinspectable macro image of relevant regions of the slide of interest.

In order to provide the magnified image, the mouse may be moved toidentify a marker segment or region which, for instance, may be arectangular region which will cause the microscope to changemagnification as at step 208 to 4×, 20×, 40×, etc., by rotating theturret to bring the appropriate objective lens system into viewingposition.

Next the user, in a step 209 a, uses the mouse to select the region onthe macro image in order to select the micro image to be viewed on thescreen 22. In a step 209 b a test is made to determine whether the userhas commanded continued inspection. If the user has, a test is made in astep 209 c to determine if the magnification is to be changed bychanging the selected objective. In the event the magnification is to bechanged control is transferred to the step 208. If the magnification isto remain unchanged control is transferred to the step 209 a. In theevent inspection is not to continue the region selected is outlined forhigher magnification scan in a step 209 d. In a step 209 e, a commandmay be received to scan or acquire the higher magnification image fordisplay in screen 26. The image may then be archived for later analysis,displayed or analyzed immediately.

In order to perform the magnification called for in step 208, theoverall illumination and control of the microscope will be controlled sothat in a step 210 the objective turret 142 will be rotated to place thehigher power objective above the slide 18. In a step 212 voltage to thelamp will be changed to adjust the lamp 168 to provide the properillumination and color temperature as predetermined for the selectedobjective. In a step 214, the condenser diaphragm 176 will have itsopening selected as appropriate to provide the proper illumination forthat objective. In a step 216, the filter turret 180 will select theproper light wavelength filter to be supplied to the camera sensors. Forinstance, a red, blue or green filter, as appropriate, particularly ifthe specimen has been stained. In a step 218 the field diaphragm 174will have its opening changed. In a step 220 the neutral density filterwheel 170 will select a neutral density filter and in a step 222 theneutral density filter wheel 172 will also select a neutral densityfilter. In a step 224 the X, Y and Z offsets will be used forreconstruction of the recorded image at the magnification and in a step226 the current position will be read from encoders in the stage whichare accurate to 0.10 micron.

In order to identify the selected region the mouse is moved to that areaof the region in a pointing operation in a step 240 as shown in FIG. 14.The mouse may be moved to draw a box around the region selected. In astep 242 the X and Y screen points are computed for the edges of theregions selected and the computed image or pixel points are translatedto stage coordinate points in order to control the stage of themicroscope. In a step 244 a list of all of the X fields for positioningthe stage for the objective is stored in random access memory and may bebacked up on the hard disk. The information from the X offsets for theobjective and the stage offsets is used as well as the size of the fieldto position the slide properly under the objective to capture the microimage.

When the slide has been positioned properly, as shown in FIG. 15 in astep 250 the stage is positioned for each of the X and Y coordinatevalues in stage coordinate values and the digitized image is captured bythe cameras and stored in RAM and backed up on the hard disk. The imagemay be then analyzed quantitatively in various manners such as those setforth in the previously-identified United States application. Optionallythe image may be stored for archival purposes in a step 254.

In order to override the specific control functions that take place asshown in FIG. 12, a screen is provided as shown in FIG. 13 wherein theX-Y step size can be edited, the X, Y and Z offset can be edited, thelamp voltage can be selected, the neutral density filter can be selectedas well as the opening of the field diaphragm and several othermicroscopic characteristics. FIG. 13 is a view of the settings of themicroscope objective properties of the Axioplan 2, computer-controlledmicroscope.

The X and Y positioning is specifically carried out as shown in FIG. 16where the slide 18 is shown with a slide boundary 270, 272, 274 and 276.Stage boundary for limits of the stage travel for purposes of the stagethe stage can be moved all the way from an upper left hand corner oftravel 276 to a lower right hand corner of travel 280. At the upper lefthand bounded corner of travel 278 limits which a signal that the end oftravel has been reached and the stage is then translated a shortdistance 282 in the extra action and a short distance 284 in the Ydirection to define the first tile 288 in terms of a reference point 290at its upper left hand corner. Since the size of the macro image tile288 is known, the next macro image tile 292 may be placed contiguouswith it by moving the stage appropriately and by measuring the locationof the stage from the stage in counters without the necessity ofperforming any image manipulation. The image tiles 288 and 292 may beabutted without any substantial overlap or they may be overlappedslightly, such as a one pixel with overlap, which is negligible insofaras blurring of any adjacent edges of abutted image tiles. The upper lefthand corner 300 of the tile 292 defines the rest of 292 and other tilescan be so defined. Micro image tiles can likewise be defined so thatthey are contiguous but not substantially overlapping, as wouldinterfere with the composite image. This avoids the problems encounteredwith having to perform extended computations on digital images in aframe storer or multiple frame storage in order to match or bring theimages into contiguity without blurriness at the edges of contiguousimage tiles. It may be appreciated that the low power image 24 has aplurality of micro images defined therein which are tiled and which areshown in higher magnification as individual tiles 312, 314, 316 and thelike. In addition, the region 310 when magnified as shown in the window26 may exceed the bounds of the window and thus the window may includescroll bars or other means for allowing the image 310 which is largerthan the window 26 to be examined from within the window 26.

The stage 200 is best seen in FIG. 16A and includes the X and Y steppermotors 279 and 281 with their respective encoders, which provide aclosed loop system to give the 0.1 micron accuracy versus the usual 5 or6 micron accuracy of most microscope stages without a closed loopsystem. This closed loop system and this very high accuracy allow theabutting of the tile images for both high magnification and lowmagnification images without the substantial overlap and thetime-consuming and expensive software currently used to eliminate theoverlap and blurriness at the overlapping edges of adjacent image tiles.With the precisely positioned stage and by using the tiling systemdescribed in connection with FIG. 16, where the slide is preciselypositioned relative to a center point CP for the slide, and the knownposition of point 278 is always taken from the same point, the tiles maybe positioned precisely in a horizontal row and precisely in verticalrows to reconstruct the macro image and the micro image. Thisreconstruction is done without the use, as in the prior art, ofextensive software manipulation to eliminate overlapping image tiles,horizontally or vertically or the haphazard orientation of image tiles.

The present invention also includes the facility for allowing remoteobservation to occur by being able to couple the system either over anetwork communication facility to an intranet, for instance via thenetwork interface, or via a modem or other suitable connection, to aninternet so that once the image has been scanned and stored in memory onhard disks or other storage, remote users may be able to access the lowmagnification image as well as the high magnification image and movearound within both images to make determinations as to the histologicalcharacteristics of the samples.

An additional feature of the system includes a plurality of networkedworkstations coupled to a first computer console 12 having a displayscreen 22 connected to the microscope 14. Satellite work stations 350and 352 are substantially identical to the work station 12 includingrespective computers 354 and 356 coupled to displays 358 and 360. Thedevices can be manipulated through input devices 360 and 362 which mayinclude a keyboard, mouse and the like. Also a third device can beconnected including a work station 370, having a display 372, a computer374 and an input device 376. Each of the devices is connected overrespective network lines 380, 382, 384 to the computer 12 whichtransmission may be via either net or the like. Each of the differentoperators at the physically separate viewing stations can locate regionsfrom the view of entire tissue cross section via a macro view and labelthe regions for subsequent scanning and/or quantitative analysis. Asingle operator at the instrument station 12 can locate regions to viewthe entire tissue cross section. Those regions can be labeled forsubsequent scanning and/or quantitative analysis with subsequent reviewand physically remote viewing stations, for instance, in an operatingroom or in individual pathologists' signout areas in order to reviewanalysis results while still maintaining and reviewing the entire macroview of the tissue and/or the individual stored images from which thequantitative results were obtained. The viewing stations 350, 352 and370 can comprise desk top computers, laptops, etc. There is no need fora microscope at the network stations 350, 352 and 370.

In a still further alternative embodiment, remote workstations 400, 402,404, 406 and 408 may be connected through a server 410 which may besupplied via a packet switched network. The server 410 and may be ahypertext transport protocol based server of the type used for the WorldWide Web or may be a telnet type server as used previously in internetremote operation applications. The server 410 communicates via acommunications channel 414 with a local computer 416 having a display418 associated therewith, the local computer 416 being connected to themicroscope 420. Each of the remote work stations 400, 402, 404, 406 and408 may perform the same operations as the stations 350, 352 and 370although they do it from nearby buildings or even from around the world,thus providing additional flexibility for others to make use of thespecimen obtained and being viewed under the microscope 420. Inaddition, stored images may be disseminated through the server 410 tothe remote servers 400 through 408 for further analysis and review.

While there has been illustrated and described a particular embodimentof the present invention, it will be appreciated that numerous changesand modifications will occur to those skilled in the art, and it isintended in the appended claims to cover all those changes andmodifications which followed in the true spirit and scope of the presentinvention.

1. A virtual microscope slide image data structure derived from a smallspecimen on a microscope slide comprising: stored image data beingstored in a computer readable storage medium and derived from a seriesof digitized magnified images of adjacent segments of the specimen takenthrough a magnifying lens, the stored image data being useable toprovide a magnified, overall digital image of at least a portion of thesmall specimen for viewing; stored image data being stored in a computerreadable storage medium and being useable to provide a digital image ofa smaller area within the specimen portion for viewing the smaller areaat least one higher magnification than that of the overall magnifiedimage of the specimen portion; and identifying data being stored in acomputer readable storage medium and correlating the magnified overalldigital image and respective digital images at a multiple highermagnification of a selected specimen area within the specimen portion.2. A virtual microscope slide data structure in accordance with 1wherein the stored image data is stored in a computer readable storagemedium and comprises: magnified images having stained, treated orenhanced features of the specimen of sufficient resolution to beanalyzed.
 3. A virtual microscope slide data structure in accordancewith claim 1 wherein the stored images provides magnified images ofsufficient resolution to be viewed and analyzed with respect to disease.4. A virtual microscope slide data structure in accordance with claim 1wherein the identifying data is stored in a computer readable storagemedium and correlates: the higher magnification images of adjacent viewsat the same resolution for scrolling into view to slow neighboringportions of the specimen; and multiple higher magnification images for aspecimen area to provide zooming between multiple resolutions for thearea of the specimen.
 5. A virtual microscope slide data structure inaccordance with claim 1 comprising: data being stored in a computerreadable storage medium and providing a dynamic program effective forviewing and manipulating into view a magnified image of the specimensent over the Internet.
 6. A virtual microscope data structure inaccordance with claim 5 wherein the stored data for the programs isstored in a computer readable storage medium and comprises: aself-executing program.
 7. A virtual microscope data structure inaccordance with claim 6 wherein the self-executing program is a JavaApplet.
 8. A virtual microscope slide data structure in accordance withclaim 1 wherein the data structure is compatible with being transmittedon the Internet.
 9. A computer readable storage medium in accordancewith claim 1 wherein the data structure comprises a stored data set oftiled image data and identifying data for an identified specimen on aslide specimen arranged for transfer over the Internet.
 10. A computerreadable storage medium in accordance with claim 1 comprises aself-contained entity storing the data for a particular specimen in amanner for distribution of the self-contained entity for the specimenover the Internet.
 11. A virtual microscope image data structure inaccordance with claim 1 stored on a computer readable storage mediumcomprising: identification data thereon identifying the microscope slidedata structure as an entity for distribution of the data structure fromthe computer readable storage medium for use as a substitute virtualmicroscope slide for the actual microscope slide.
 12. A virtualmicroscope image data structure in accordance with claim 11 stored on acomputer readable storage medium wherein the data structure is one ofseveral virtual microscope slide data structures stored as separateidentifiable entities on the computer readable storage medium fordistribution by their respective identification data.
 13. A virtualmicroscope image data structure in accordance with claim 11 wherein thedata structure is stored on the computer readable storage medium at anInternet server.
 14. A virtual microscope image data structure inaccordance with claim 13 wherein stored image data comprises tiledblocks of data for sending over the Internet.
 15. A virtual microscopeimage data structure in accordance with claim 11 wherein the datastructure is stored on the computer readable storage medium at apersonal computer.
 16. A virtual microscope data image structure inaccordance with claim 11 where the data structure is stored on aportable computer readable storage medium.
 17. A virtual microscopeimage data structure in accordance with claim 16 wherein the datastructure is stored on a disc form of portable storage medium.
 18. Avirtual microscope image data set in accordance with claim 1 wherein theidentifying data comprises coordinate system data for a selected pointon the specimen to be shown as images at different magnifications.
 19. Avirtual microscope image data structure in accordance with claim 18comprising: identification data for use in zooming between displayimages of the same point at different magnifications and for scrollingan adjacent image into view on a display device.
 20. A virtualmicroscope image data structure in accordance with claim 1 wherein thestored image data of adjacent magnified image segments comprises theimage segments that capture the entire specimen on the slide.
 21. Amethod of constructing a virtual microscope slide data structure derivedfrom a specimen or a portion thereof on a microscope slide comprising:using a magnifying lens and digitally scanning magnified images ofsegments of the microscope specimen; storing the digital data for thedigitized images of magnified segments of the specimen which digitaldata is useable to provide an overall magnified image which is larger inarea than a single optical magnified image of a segment and whichdigital data is useable to provide higher magnification images of theselected area at the higher magnification, wherein the scanningmagnified image of segments and storing digital data for digitized imagesegments capturing the entire specimen on the slide as the overallmagnified image; and using identifying data to coherently assemble thesegment data thereof in order to provide an overall magnified image ofthe specimen.
 22. A method in accordance with claim 21 wherein the useof a magnifying lens comprises: using a microscope objective lens onspecimens having features so small that they cannot be effectivelyviewed and analyzed without magnification.
 23. A method in accordancewith claim 21 comprising: providing a stained or otherwise treatedspecimen to contrast different characteristics of the specimen forviewing.
 24. A method in accordance with claim 21 wherein the digitallyscanning the specimen comprises: performing a first scanning and storingdigital data taken at a first low magnification through the magnifyinglens; displaying an overall low magnification image to a viewer;selecting for scanning a region on the displayed overall image forscanning; and performing a second scanning operation of the selectedregion at a higher magnification than the first magnification andstoring of the data for the higher magnification image of the selectedregion.
 25. A method in accordance with claim 24 further comprising:storing the data structure on a storage medium at an Internet server fortransmission therefrom.
 26. A method in accordance with claim 21comprising: constructing the data structure with programs to identifythe stored segments to enable them to be coherently assembled in orderto provide the overall image of the specimen portion and to enableviewing of high magnification images of the selected areas.
 27. Themethod of claim 26 comprising: providing the program as a Java applet.28. The method of claim 21 comprising: constructing the data structurewith a dynamic, self-executing program effective for viewing andmanipulating into view the magnified image.
 29. The method of claim 21comprising: string on the data structure coordinate information forenabling the coherent assembly of the adjacent, digitally scannedsegments.
 30. The method of claim 21 comprising: compressing thedigitized images and storing the compressed digitized images.
 31. Themethod of claim 21 wherein the digitally scanning comprises: scanningeach of the digitized image segments as an image which is substantiallyan optical view as seen through an objective lens of a scanner.
 32. Amethod in accordance with claim 21 wherein the identifying data furthercomprises: identifying the specimen data structure as entity fordistribution of the data structure to a display device for viewingimages of the identified specimen.
 33. A method in accordance with claim32 comprising: storing of the data structure as one of several separateidentifiable microscope specimen entities stored on a storage medium andusing respective identifying data for each of them for distribution ofeach of them.
 34. A method in accordance with claim 21 furthercomprising: storing the data structure on the storage medium at apersonal computer.
 35. A method in accordance with claim 21 furthercomprising: storing the data structure on a portable storage medium. 36.A method in accordance with claim 35 further comprising: storing thedata structure on a disc form of portable storage medium.
 37. A methodin accordance with claim 21 further comprising: transmitting the datastructure as tiled data blocks over the Internet.
 38. A method inaccordance with claim 21 further comprising: providing identifying datacomprising coordinate system data for use in displaying a selected pointon the specimen to be shown as magnified images at differentmagnifications.
 39. A method in accordance with claim 38 comprising:zooming between displayed images of the same point at differentmagnifications and scrolling a neighboring image into view on a displaydevice.
 40. A method in accordance with claim 21 wherein the stored datais flexibly expanded to provide images for larger sizes or shapes ofspecimens.
 41. A method in accordance with claim 21 wherein theidentifying data comprises an organizational structure which iscontained in the virtual microscope data structure for enabling coherentassembly of the segment data to provide specimen images.
 42. A method inaccordance with claim 41 wherein the organizational structure comprises:stage coordinate information for coordinating positioning betweenmultiple resolution images for the same point on a specimen.
 43. Amethod in accordance with claim 21 comprising: using a magnifying lensduring the digital scanning to obtain high resolution and highermagnification images to allow optical zooming to achieve lowermagnification images with good fidelity.
 44. A method in accordance withclaim 21 wherein the digital scanning comprises moving the specimenrelative to the magnifying lens with steps smaller than a pixel sizebeing used for recording the image.
 45. An apparatus for creating avirtual microscope data structure comprising: a computer-controlledmicroscope imaging system having an objective lens for digitallyscanning a series of magnified image segments from a specimen on asupport; a program means for recording the scanned digital, magnifiedimage segments; a program means for storing the image segments; aprogram means for linking the series of contiguous image segments with aprogram effective for providing an overall magnified image of thespecimen; a program means for linking higher magnification images of theselected areas of the specimen with the overall image for display; and adata compressor means for compressing the data of the digitally scannedsignificantly to allow the scanned digitized images to be sent over theInternet.
 46. An apparatus in accordance with claim 45 wherein theprogram for linking the images comprises: an addressable coordinatemeans for addressing the image segments for being scanned together andfor addressing magnification images.
 47. An apparatus in accordance withclaim 46 wherein the program means for linking the images comprises: aprogram for scrolling and allowing a viewer to scroll a portion of aneighboring image into view.
 48. An apparatus in accordance with claim45 wherein the program means comprises: an address display means fordisplaying the coordinates of an image to assist multiple viewers toidentify the same area for analysis and commentary.
 49. An apparatus inaccordance with claim 45 further comprising: a browser means for storingthe dynamic, self-executing program; and a monitor means for viewing theimages stored on the monitor and for flipping back and forth between adifferent high magnification, micro images of selected areas of thespecimen.
 50. An apparatus in accordance with claim 49 comprising: amarker program means for marking an addressable area on the macro imageand to cause the addressed area to appear as a higher resolution, microimage of the specimen on the monitor.
 51. A method of constructing avirtual slide data structure derived from a specimen or portion of aspecimen on a microscope slide and storing the data structure at a webserver comprising: capturing and storing image data for an overallmagnified image of a specimen and higher magnification images selectedportions of the specimen; storing digital image data on a storage mediumat a web server which is useable to provide a low magnification overallimage of the specimen; and storing digital image data on the storagemedium at the web server useable to provide higher magnification imagesof a specimen area which is smaller than an overall specimen area; andidentifying stored higher magnification images at differentmagnifications for selective display of the same point at differentresolutions.
 52. A method in accordance with claim 51 comprising:constructing the data structure to enable viewing by taking the storedimage data and converting it with a program for viewing and manipulatingimages at the different magnifications.
 53. A method in accordance withclaim 51 comprising: storing a program on the storage medium fortransfer with the image data to a remote display means.
 54. A method inaccordance with 51 comprising: accessing the data structure andtransferring image data therefrom for use with a program stored at theremote site to display and manipulate magnified images.
 55. An apparatusfor constructing a virtual microscope slide data structure derived froma specimen or a portion thereof on a microscope slide comprising: amicroscope imaging system having a magnifying objective lens fordigitally scanning adjacent segments of the specimen to capture digitaldata of magnified images of the microscope specimen useable fordisplaying an overall image of the specimen and for displaying a highermagnified image of a selected area of the overall image; a storagemedium for storing the digital data for the digitized images of adjacentsegments of the specimen which digital data is useable to provide anoverall magnified image which is larger in area than a single opticalmagnified image of a segment and which digital data is useable toprovide higher magnification images of the selected area at the highermagnification; and a program identifying the stored segment images toenable a coherently assembly of the segment images in order to providean overall magnified image of the specimen.